Data storage vendors offer a wide range of data storage systems. When new features or other changes are made to a data storage system, thorough testing is performed to maintain outstanding storage quality. For example, at each release development cycle, endurance testing may be performed. One goal of endurance testing is to verify that the content of stored objects is not corrupted with a lapse of time. To facilitate endurance testing, a testing tool may generate many large data objects (further referred to herein as “objects”), store these objects within the storage system, and subsequently read the objects back from storage and verify their contents. In between creating and verifying a given object, a large number of I/O operations may be performed over some relatively long time period, including creating and deleting other objects.
To verify stored object content, a common approach is to calculate a checksum over the object's data and to store the checksum locally, such as to a local disk or other type of non-volatile memory, along with an object ID. When the object is read back, a second checksum is calculated over the stored object data and compared with the locally stored checksum. If the checksums are equal, the object content is verified. Otherwise, the object is reported as corrupted. Because checksums are not one-to-one functions (i.e., the same checksum can be computed for different data), object content verification using checksums is not 100% reliable. In addition, checksums cannot be used to determine the specific data within the object that was corrupted, complicating root cause analysis.
Another approach is to store a replica of each object locally. Although this approach is reliable, it is typically impractical due to the high storage overhead.
In addition to storing new objects, object storage systems may support modifying the contents of existing objects.